feat(api-proxy): offer two priced models instead of one - #72
Merged
Conversation
The README claimed a second model was "more config, not code" without showing it. Run nginx.conf.template twice with a different MODEL, give each its own runners.json entry, app id, and price, and let the client pick with --app. FLUX.1 schnell is cheaper than SD3 medium, so the two prices differ for a reason a caller can see. Each entry keeps its own capacity: the work happens upstream, so the capabilities do not contend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
Two containers implied a capability is a process. It is a registration: one nginx now carries a route per model, and each runners.json entry points its app id at its own path, which the orchestrator preserves when it forwards. Keeps the repo consistent with ollama, where one container serves several capabilities. Health stays shared. `return 200` only reported that nginx is up, never that a model or the token is good, so a copy per capability would claim a precision it does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
rickstaa
force-pushed
the
rs/api-proxy-two-models
branch
from
August 17, 2026 10:14
f3bc8a8 to
0304f5e
Compare
The MODEL_* variables only moved a literal from one file to another, and swapping a model costs the same two edits either way. Pinning the URLs in nginx.conf.template puts each model beside the route that serves it, and makes "the pinned URL is the security model" true of the file you read rather than of the environment at runtime. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
Hugging Face exposes no readiness endpoint, only metadata, and the orchestrator polls health every 5s with a 3s timeout while a failed poll also releases live sessions. Record that in the comment so the next reader does not repeat the investigation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
The on-chain run section re-argued the point the capability section already makes, inside a passage that should only say what to type. Keep the signer cap, which is the part unique to running on-chain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
The health reasoning now lives in nginx.conf.template, where it also records that Hugging Face exposes no readiness endpoint, so the README points at it rather than restating a weaker version. The static versus dynamic section loses its closing summary, which only repeated the two paragraphs above it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
capacity was 1, so three of four concurrent callers got a 503 while nothing on the machine was busy: the work happens upstream and nginx only holds a socket. Measured, four concurrent calls all get through at 4. For a proxy capacity is exposure control, capping paid-but-unfulfilled calls against the operator's credential, so it belongs to the upstream quota rather than to this machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The api-proxy README claimed that offering a second model is "more config, not code", but the example only ever showed one. Now it shows two, served from the one nginx.
nginx.conf.templategets a route per capability,/sd3/proxyand/flux/proxy, each pinning its own model.runners.jsongets a second entry with its own app id and price, each pointing at its own path (http://app:8989/sd3), which the orchestrator preserves when it forwards. FLUX.1 schnell is faster and cheaper than SD3 medium, so the two prices differ for a reason a caller can see.client.pygains--app, which is what actually demonstrates that discovery filters on app id, matched exactly.livepeer/api-proxybecomes a section: the static versus dynamic contrast only lands once there are two pinned capabilities to hold against runtime enablement.One process, two registrations, because a capability is a registration and not a container.
Verified offchain end to end: both register healthy, both appear in
/discoveryas separate capabilities, and a call through the orchestrator to each app id reserves a session and lands on the right pinned model. The upstream returns 401 on a dummy token, which is as far as this goes without anHF_TOKENcarrying inference-provider credits, so no real generation was run.